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PSYCHOPHYSIOL OGICAL  EFFECTS  OF  ACING  - DEVELOPING  A FUNCTIONAL  AGE 
INDEX  FOR  PILOTS:  III.  MEASUREMENT  OF  PILOT  PERFORMANCE 

Introduction. 


In  two  earlier  reports  on  this  subject,  a literature  survey  and  a 
taxonomy  of  psychological  factors  which  are  age-related  and  essential  to  pilot 
performance  were  presented  (13.14).  It  was  observed  that  the  14  factors, 
identified  by  our  taxanomic  survey  do  meet  the  basic  criteria  of  theoretical 
and  operational  applicability  in  regard  to  the  assessment  of  aviator 
proficiency  (7).  We  also  concluded  from  our  previous  work  that  there  are 
performance  differences  between  younger  and  older  pilots  and,  based  on  avail- 
able statistical  criteria,  that  the  rules  which  govern  the  statistical 
distribution  of  abilities,  skills,  and  the  underlying  psyehophysiological 
functions  may  or  may  not  wcrk  in  individual  eases.  It  is  well  known  that 
individuals  who  are  of  the  same  chronological  age  differ  significantly  as  to 
their  functional  or  performance  capabilities-  Any  attempt  to  develop  a 
functional  age  index  for  pilots  must,  therefore,  deal  with  the  means  and 
methods  available  to  measure  group  and  individual  pilot,  performance. 


We  would  like  to  point  out  that,  based  on  statistical  data  published  over 
the  years  by  the  National  Transportation  Safety  Board  (NTSB),  performance  and 
performance  failures  appear  to  be  more  important  to  safety-related  pilot 
proficiency  than  are-  health  or  medical  disability  in  flight.  The  number  of 
fatal  and  nonfatal  general  aviation  accidents,  in  which  the  pilot-in-command 
is  listed  as  the  cause  cr  a contributing  factor  during  the  5-year  period  from 
197(j  to  iy?4,  is  shown  in  Table  1.  In  analyzing  these  data,  Denson  and  Benel 
of  the  University  of  Illinois  (23)  established  three  behavio  al  categories, 
namely,  Procedural  Activities,  Perceptual  ■•Motor  Activities,  and  Decisional 
Activities,  and  they  included  accidents  w iich  involved  medical  causative 
factors  into  this  last  category  (factors  numbered  23  and  24  in  the  table). 
After  summing  the  incidences  for  these  latter  two  factors,  we  find  that  they 
account  for  less  than  5 percent  of  the  total  fatal  and  less  than  one-fourth 
of  1 percent  of  all  nonfatal  accidents  (25).  One  reason  for  this  particular 
relationship  observed  in  general  aviation  may  be  that  the  private  pilots  must 
be  medically  examined  and  certified  at  regular  intervals,  whereas  there  are 
no  regular  performance  checks  required.  But  the  dominance  of  nonmedical  human 
factors  over  medical  factors  also  exists  in  air  carrier  accidents  in  which 
illness  and  sudden  physical  incapacitation  of  the  pilot  play  a relatively 
minor  part  (33).  This  makes  the  analysis  and  measurement  of  pilot  performance 
an  even  more  important  issue. 


Research  on  Aviator  Performance. 
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There  has  been  extensive  research  on  aviator  performance  determinants  as  "**V 

part  of  the  various  aviation  nsychology  programs  in  this  country  and  abroad. 
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TABLF  1 Nimber  of  fatal  and  Nonfatal  General  Aviation  Accidents  in  Which 
the  Pilot  in  Commend  la  Listed  as  the  Cause  or  a l actor  for  all  Data 


Retween  1970  and  1974  for  Three  Behavioral  Categories 

(23) 

5-Year 

Totals 

Procedural  Activities 

Fatal 

Monfatal 

1. 

failed  to  extend  landing  gear 

1 

255 

2. 

failed  to  retract  landing  gear 

4 

14 

3. 

Failed  to  use  or  incorrectly  used  miscellaneous  equipment 

14 

6? 

4. 

Improper  IFR  operation 

110 

66 

5. 

Improper  fuel  management 

105 

1,231 

6. 

Improper  Starting  procedure 

1 

30 

7. 

Failed  to  assure  gear  down  and  lacked 

1 

207 

8. 

Misused  or  fa 'led  to  use  flaps 

27 

235 

9. 

Inadvertently  retracted  landing  geci 

0 

104 

10. 

Retracted  gear  prematurely 

1 

26 

Total  for  Procedural  Activities 

264 

7,2  30 

Percent  of  total  pilot-caused  accidents 

4.6 

9.6 

Perceptual-Motur  Activities 

1- 

Delayed  action  in  aborting  takeoff 

5 

236 

2. 

Delayed  In  Initiating  go- around 

32 

380 

3. 

Failed  to  see  and  avoid  other  aircraft 

128 

196 

4. 

Failed  to  see  and  avoid  object 

IGG 

757 

5. 

lolled  to  maintain  flying  speed 

846 

1,825 

6. 

Misjudged  distance,  speed,  altitude,  clearance 

35i 

2.H64 

7. 

Failed  to  maintain  adequate  rotor  RPM 

16 

153 

8. 

Improper  operation  of  power  plant  eonvrols 

51 

685 

9. 

Iafiroper  operation  of  brakes/flight  controls 

1 

688 

10. 

Improper  operation  of  flight  cent; jls 

i64 

569 

11. 

Improper  level-off 

10 

1,596 

12. 

Improper  compensation  for  wind 

12 

550 

13. 

Control  interference 

n 

1 

14. 

Improper  recovery  from  bounced  landing 

5 

811 

15. 

Spatial  disorientation 

528 

60 

18. 

failure  to  maintain  directional  control 

13 

1,978 

17. 

Premature  liftoff 

11 

302 

18. 

Tailed  to  abort  takeoff 

26 

257 

19. 

failed  to  Initiate  gn-aroi«nd 

ft 

637 

20. 

Exceeded  design  stress  limits  of  aircraft 

121 

16 

Total  for  Percepcual -Motor  Activities 

? ,4V* 

H 5 c l 

Percent  of  total  pilot-caused  accidents 

43.8 

56.3 

Decisional  Activities 

1. 

Operation  of  aircraft  with  known  deficiencies 

84 

201 

2. 

Operation  beyond  experlencc/abi 11 ty 

170 

369 

3. 

Continued  VT II  into  known  adverse  weather 

717 

343 

4. 

Continued  flight  Into  known  severe  turbulence 

18 

7 

5. 

Improper  inflight  decisions/planning 

2 36 

597 

6. 

Exercised  p.wr  Judgment 

235 

767 

7. 

Operated  carelessly 

7 

38 

8. 

Selected  unsuitable  terrain 

22 

1 ,230 

9. 

Initiated  flight  into  adverse  weather 

if 

61 

10. 

Psychological  condition 

U 

4 

11. 

Selected  wrong  runway 

11 

341 

12. 

failed  to  follow  approved  procedures 

145 

425 

13. 

Inadequate  preflight  planning  or  preparation 

511 

2,341 

14. 

Lack  of  familiarity  with  aircraft 

121 

611 

15. 

Started  without  proper  assistance 

6 

89 

1C. 

Became  Jnst/di xnrientert 

74ti 

17. 

Taxied,  parked  without  proper  assistance 

0 

6 7 

18. 

Left  aircraft  unattended 

1 

8 

19. 

Diverted  attention  from  operation  of  aircraft 

111 

50] 

20. 

Inadequate  supervision  of  flight 

6.’ 

610 

21. 

Spontaneous  improper  action 

15 

119 

22. 

Misunderstood  orders/ ins true lions 

3 

20 

23. 

Incapacitation 

50 

7 

24. 

Physical  impairment 

203 

65 

25. 

Inadequate  training 

0 

5 

26. 

Direct  entry 

9 

14 

Total  for  Decisional  Activities 

>,940 

9,(^7 

Percent  of  total  pi  lot -caused  accidents 

51.6 
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Historically,  interest  in  the  assessment  of  pilot  proficiency  dates  back  to  1 
the  work  on  military  aviation  problems  during  World  War  I.  This  effort  was  | 
greatly  accelerated  in  World  War  II,  and  it  continues  at  this  time  by  ? 
generally  following  the  methodological  principles,  techniques,  and  operational  | 
procedures  of  the  earlier  period.  Generally  speaking,  performance  .'as  been  ’ 
assessed  against  a definite  task  specification  that  had  been  obtained  by  J 
either  operational  analysis,  subjective  judgments  by  experts  in  this  3 


particular  field,  or  numerous  performances  sampled  from  adequate  populations  a 
(.lb).  There  are  two  major  approaches  in  which  pilot  performance  assessment  | 
can  be  categorized.  The  earliest  method  used  in  aviation  was  the  qualitative  | 
evaluation  of  performance  based  on  subjective  ratings  bv  flight  instructors  or  J 
inspectors,  flight  examiners,  or  check  pilots.  Today,  the  rater  may  use  some  % 


form  of  quantitative  verification  technique  such  as  descriptions  of  action  | 

taken,  record  sheets,  or  quantitative  rating  scales  or  score  cards.  1 


The  second  method  of  performance  assessment  consists  of  the  objective  J 
and/or  automatic  recording  of  the  major  performance  criteria  and  evaluation  s 
against  standardized  criterion  measures-  The  goal  of  this  effort  is  to  arrive  | 
at  an  objective  system  that  leaves  no  margin  for  human  error.  At  present  the  J 
method  most  commonly  used  consists  of  various  mixed  techniques,  whereby  the  | 
subjective  ratings  of  an  observer  are  complemented  and  correlated  with  the  j 
data  obtained  by  an  objective  recording  system  or,  vice  versa,  where  these  two  J 
methods  are  designed  to  supplement  each  other.  In  this  way,  more  complete  | 
information  on  pilot:  performance  in  a more  or  less  realistic  situation  can  be  1 
obtained. 

■J 


As  part  of  a feasibility  study  dealing  with  the  automated  performance 
assessment  of  military  pilots,  Knoop  and  Welde  (26)  discussed  the  significant 
problems  inherent  in  the  development  of  an  objective  pilot  performance 
measurement  system.  They  rightly  point  out  the  many  difficulties  involved  in 
such  an  attempt.  In  accordance  with  the  concept  described  by  Glaser  and 
Klaus  in  1963,  they  consider  the  environment  in  which  performance  is  measured 
as  a major  source  of  variability  (16).  Other  sources  of  variance  are  the 
fluctuations  inherent  in  the  system  that  is  used  to  measure  performance. 
Sensors,  sample  selection,  software,  system  operators,  and  response-evaluating 
instruments  contribute  to  system  variability. 


Of  the  human  factors  directly  involved  in  performance  measurement,  the 
complexity  of  the  behavior  being  evaluated  and  the  individual  differences 
affect  the  consistency  and  reliability  of  the  measures.  Since  an  individual's 
performance  level  may  change  measurably  from  one  occasion  and  one  dimension 
to  the  next,  each  component  element  in  a sense  represents  a new  condition  of  a 
somewhat  different  level  of  difficulty.  Also,  the  psychological  and  physio- 
logical conditions  of  the  pilot  himself  are  a source  of  performance  variations, 
but  we  must  assume  a certain  amount  of  system  stability  or  homeostasis  in  our 
measurement  process.  Even  so,  the  variations  in  the  scores  or  data  obtained 
do  reflect  a certain  degree  of  bias  and  random  fluctuations  caused  by  system 
instability,  intra-individual  variability,  and  other  remnant  factors. 
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Conceptually,  performance  measurements  of  the  kind  we  are  interested  in 
must,  regardless  of  the  degree  of  subjectivity  involved,  therefore,  be 
designed  to  minimize  or  eliminate  fluctuations  and  variability  to  produce 
reliable  results.  Of  primary  importance,  as  formulated  by  Knoep  and  Weide 
(25)  is  the  necessity  to  apply  realistic  conditions  and  criteria  in  the 
measurement  of  pilot  performance,  so  that  the  technique  and  the  results 
obtained  are  accepted  by  the  pilot. 

In  1952,  Smith,  Flexman,  and  Houston  of  the  Human  Resources  Research 
Center,  Air  Training  Command,  developed  a technique  for,  as  they  called  it, 
"objectively"  recording  pilot  performance  (35).  They  admitted,  however,  that 
the  "Performance  Record  Sheets"  which  were  used  in  the  experiment  were 
designed  to  describe  but  not  to  rate  pilot  performance.  It  was  thought 
essential  to  develop  procedures  which  would  permit  recording  Inflight  perform- 
ance and  to  allow  for  reliable  descriptions  which  could  be  repeated  by  several 
flight  observers.  The  first  step  in  this  procedure  was  to  examine  all 
maneuvers  required  in  the  Primary  Training  Syllabus  and  to  break  down  each 
maneuver  into  its  components.  This  item  breakdown  was  accomplished  by  a team 
of  flight  instructors  and  psychologists  and  aimed  at  the  isolation  of  the 
critical  flight  elements. 

The  Performance  Record  Sheets  mentioned  before  were  then  tried  out  on  the 
specified  maneuvers  to  assure  that  the  record  procedure  was  efficient  arid 
practical.  In  addition,  observer  reliability  studies  were  conducted  to 
determine  the  degree  of  agreement  between  the  two  instructors  who  observed  the 
same  pilot  performance.  There  were  two  direct  products  of  this  effort: 

First,  the  maneuver  analysis  was  made  tc  cover  all  important  pilot  activities 
and  second,  the  technique  was  rendered  reliable  and  standardized  for  obtaining 
pilot  proficiency  measures.  The  authors  concluded  that  this  research  repre- 
sented the  first  successful  attempt  to  minutely  describe  and  "objectively" 
record  actual  performance  for  both  contact  and  instrument  maneuvers. 

Subjective  Pilot  Performance  Assessment. 

Pilot  performance  assessment  is  required  by  law.  At  present,  in 
accordance  with  Part  61  of  the  Federal  Aviation  Regulation  (FAR),  tne  appli- 
cant for  a civil  pilot  certificate  must  pass  the  appropriate  written  and 
practical  tests  and  medical  examination,  must  have  the  necessary  flight, 
instructions  ana  in  the  case  of  a request  for  ar,  air  transport  rating,  be 
able  to  perform  satisfactorily  a line  check  which  includes  the  duties  anu 
responsibilities  as  specified  in  FAR  121.440.  His  ability  to  perform  the 
required  pilot  operations  is  generally  judged  by  the  way  he: 

1.  Executes  procedures  and  maneuvers  within  the  aircraft's  performance 
capabilities  and  limitations,  including  tie  use  of  the  aircraft's  system  or 
systems; 


2.  Executes  emergency  procedures  and  maneuvers  appropriate  to  the 
aircraft; 
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3.  Pilots  the  aircraft  with  smoothness  and  accuracy; 

4.  Exercises  judgment; 

b.  Applies  aeronautical  knowledge; 

6.  Shows  masterful  handling  of  the  aircraft  with  the  successful 
outcome  of  the  procedure  or  maneuver  never  seriouslv  in  doubt. 

The  syllabus  or  scenario  Oi  the  inflight  performance  check  (which  can  be 
partially  taken  in  an  approved  flight  simulator)  varies,  of  course,  in 
accordance  with  the  type  of  certificate;  but  it  contains  such  items  as 
preflight  preparations,  aircraft  performance  analysis,  handling  of  the 
aircraft  on  the  ground  and  in  the  air,  compliance  with  safe  operation 
procedures,  checklists,  and  so  on. 

The  flight  instructor,  examiner,  or  inspector  who  conducts  the  pilot 
operations  or  flight  tests  or  the  proficiency  check,  judges  or  rates  the 
applicant  in  accordance  with  acceptable  performance  guidelines.  These  guide- 
lines include  the  factors  which  will  be  taken  into  account  by  the  examiner  in 
deciding,  whether  the  applicant,  student,  or  pilot  being  checked  has  met  the 
objective  of  the  intended  operation.  Emphasis  is  placed  on  knowledge, 
procedures,  and  maneuvers  which  are  most  critical  to  a safe  performance  as  a 
pilot.  For  example,  the  demonstration  of  fast  stall  recognition,  adequate 
control  action,  and  recovery  techniques  receive  special  attention.  Other 
areas  of  importance  include  spatial  orientation,  collision  avoidance, 
vigilance,  and  wake  turbulence  hazards. 

The  Practical  lests  Guide  for  Airline  Transport  Pilots  (FAA  AG-61-A9) ( ] 1 ) 
contains  a few  remarks  about  the  rating  procedure.  It  states  that  throughout 
the  maneuvers,  if  appropriate,  good  judgment  commensurate  with  a high  level  of 
safety  must  be  demonstrated.  In  determining  whether  such  judgment  has  been 
exercised,  the  inspector/examiner  who  conducts  the  check  considers  adherence 
to  approved  procedures,  actions  based  on  the  analysis  of  situations  for  which 
there  is  no  prescribed  or  recommended  practice,  and  qualities  of  prudence  and 
care  in  selecting  a particular  course  of  action.  As  already  mentioned,  these 
actions  must  be  based  on  knowledge  of  the  airplane,  its  systems  and  components, 
and  compliance  with  approved  en  route,  instrument  approach,  missed  approach, 
ATC,  or  other  existing  and  applicable  procedures  (11). 

Notwithstanding  the  amount  of  thought,  experience,  and  care  that  is  and 
has  been  invested  in  the  present  pilot  rating  procedure,  one  has  to  admit  that 
it  is  subjective,  based  on  more  or  less  well  defined  and  clear  criteria, 
and--above  all --catering  to  the  concept  of  minimal  standards.  It  is  therefore 
well  worth  remembering  what  Knoop  and  Welde  (Zb)  stated  in  their  study  of  an 
automated  pilot  performance  assessment  system  developed  for  the  United  States 
Air  Force.  They  listed  the  following  sources  of  variance  in  subjective  pilot 
ratings : 
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1.  Judgments  of  this  sort  arc  made  without  reference  to  a definite 
standard  since  the  same  maneuver  may  be  flown  satisfactor tly  in  a number  of 
different  ways. 

2.  Different  standards  of  performance  are  usually  employed  due  to 
differences  in  the  examiner's  knowledge,  experience,  and  proficiency. 

3.  The  examiner's  operational  skill,  his  personal  assessment  of  the 
critical  aspects  of  the  maneuver  or  the  job,  a,.d  his  own  training  may  affect 
the  perspective  and  judgment  of  the  ratings. 

4.  The  examiners  differ  in  personal  bias  toward  the  student  or  pilot  to 
be  tested. 

5.  Raters  have  different  concepts  of  the  specific  grading  system  in 
regai d to  the  flight  parameters  involved,  the  knowledge  tested,  weights  to  be 
assigned,  and  the  range  of  the  qualitative  categories. 

6.  It  is  difficult  to  compare  actual  performance  with  the  conceptual 
performance  and  with  what  the  average  proficiency  level  should  be  at  the  time 
of  the  cheek  ride. 

Since  our  study  program  is  essentially  psychophysical  and  psychological 
in  nature,  the  behavioral  factors  should  be  pointed  out  that  Knoop  and  Welde 
(23)  assigned  to  the  examiner  for  evaluation: 

1.  Ability  to  plan  effectively. 

2.  Decision  making  capability. 

3.  Sensorimotor  coordination  and  smoothness  of  control. 

Ability  to  share  attention  and  efforts  appropriately  iri  an 
environment  of  simultaneous  activities. 

5.  Knowledge  and  systematic  performance  of  tasks. 

6.  Confidence  proportionate  to  the  individuai's  level  of  competence. 

7.  Maturity,  i.e.,  the  willingness  to  accept  responsibility,  the  ability 
to  accomplish  stated  objectives,  judgments,  and  reaction  to  stress, 
unexpected  conditions,  and  aircraft  emergencies. 

8.  Motivation  (attitude)  in  terms  of  the  manner  in  which  it  affects 
performance. 

9.  Coordination  with  others  (crew  members). 

10.  Fear  of  flying. 


11.  Motion  sickness. 

12.  Air  discipline,  i.  -*. , adherence  to  rules,  regulations,  assigned 
tasks,  and  command  authority  (25). 

These  behavioral  factors  are  in  very  close  agreement  with  the  14  factors 
which  were  identified  in  our  previous  taxonomic  survey  (14).  They  are  rather 
independently  found  ir.  studies  concerning  militaiy  or  civilian  airmen,  and 
they  arc  consistently  associated  with  successful  and  nonsucccssful  pilot 
performance  regardless  of  the  level  of  skill,  experience,  technology,  and 
automation.  The  main  problem  In  this  context  docs  not  concern  the  validity 
of  the  identified  psychological  and  psychophysiological  factors  In  measuring 
pilot  proficiency,  but  Lite  techniques,  methods,  and  means  with  which  these 
factors  can  be  assessed  with  the  least  error  variance  possible. 

There  are  many  examples  in  the  literature  about  attempts  to  improve 
subjective  rating  systems  (e.g.,  3, 13, '5).  They  mostly  deal  with  the  problem 
of  obtaining  quantitative  measures  that,  are  free  from  personal  or  emotional 
bias,  as  well  as  being  reproducible  and  permanent.  In  this  context,  Grunhofer 
and  Gerbert  questioned  the  validity  of  proficiency  records  obtained  from  pilots 
of  the  German  Air  force  (17).  Reporting  their  findings  at  the  AGARD  Conference 
on  Physical  Fitness  in  Tlying,  Including  the  Aging  and  the  Aged  Aircrew,  they 
concluded  that  only  objectively  measured  or  assessed  flying  performance 
reflects  intra-  and  interindividual  differences,  age-specific  changes  and, 
possibly,  insufficiences.  And  they  state:  "It  is  only  with  measurements  of 
tills  nature  that  we  could  diagnose  wh^n  a man  has  reached  the  point  where  he 
will  be  unable  to  compensate  for  performance  decrements  in  this  or  that 
particular  ability  and  in  a certain  flight  task,  and  where  the  reduced  degree 
of  reliability  of  inflight,  behavior  will  endanger  fly i no  safety  " 

The  authors  reflected  seriously  on  how  to  assess  significant  aspects  of 
performance  and  they  recommended,  as  a first  step,  the  upgrading  of  the  flight 
performance  ratings  from  the  two-grade  system  "Satisfactory"  and  "Unsatisfac- 
tory" to  a five-grade  flying  proficiency  statement,  which  would  be  prepared 
by  the  Wing  Commander  for  every  pilot  whenever  he  is  due  for  his  annual 
physical  examination.  Such  a system  would  differentiate  between  proficiency 
levels,  reduce  gross  errors  in  judgment,  demand  a more  analytical  approach  by 
the  rater,  and  provide  better  quantifiable  results.  It  would  also  be  suitable 
for  longitudinal  studies  and  permit  correlations  with  flying  experiences, 
training  status,  type  of  aircraft  flown,  physiological  and  psychological  data, 
and  age.  The  authors  concluded  that  in  this  way  it  may  be  possible  to 
recognize  in  time  "critical  symptoms  of  aging,"  identify  certain  "syndromes  of 
aging,"  and  determine  "Verbal tensalter,"  meaning  functional  age,  which  could 
be  used  as  a criterion  for  reassignment  or  retirement  from  flying. 

In  Holland,  Van  der  Laan  (35)  assessed  the  behavior,  of  which  human 
performance  is  a derivative,  of  99  k'LM  pilots  in  the  cockpit.  During  the 
regular  proficiency  checks,  pilot  behavior  was  graded  by  means  of  an  elaborate 
rating  scale.  An  analysis  of  the  main  factors  that  could  be  isolated  as  a 
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Figure  1.  Flight  evaluation  record  developed  ty  Hollister 
and  LaPointe  (20). 
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result  of  the  check  ride  yielded  the  following  loadings:  (i)  work  efficiency 
(r=0.42),  (ii)  emotional  stability  (r=G .23),  and  (iii)  sociability  (r-0.17). 

In  an  attempt  to  identify  and  determine  skill  degradation  in  private 
nr.d  Commercial  pilots,  personnel  from  the  Massachusetts  Institute  of 
Technology  (MIT)  conducted  flight  performance  tests  for  the  I'M  in  1972/73 
(20).  five  experienced  pilots  were  assigned  ac  evaluators  for  the  flight 
test  program  conducted  Iri  a Cessna  130  aircraft.  Their  evaluation  procedure 
was  "standardized"  on  a Flight  evaluation  Record  Form  (see  figure  1)  through 
discussion  periods,  standardized  flights,  and  the  following  guidelines: 

"Skill  grades  were  assigned  as  indicated  on  the  Flight  evaluation  Record 
Form  for  major  subareas  of  each  flight,  plus  an  overall  grade  and  written  quiz 
grade,  when  taken.  A grade  was  entered  in  all  boxes  for  which  the  subject's 
performance  was  observed  and  a dash,  if  the  box  was  not  applicable  to  the 
flight  or  the  maneuver  was  not  performed.  Grades  were  assigned  on  the  basis 
as  follows:  3 = perfect,  4 = above  average,  3 - average,  1 = unacceptable, 
and  0 = dangerous. 

"For  all  flights,  grades  were  given  on  the  following:  (1)  Aircraft 
preflight,  (2)  start,  taxi,  and  run-up,  (3)  takeoff  and  departure,  (4)  simu- 
lated engine- out , (3)  radio  procedures,  (6)  landings,  anc  (7)  overall  grade. 
For  the  first  and  last  flights,  additional  grades  were  included  on  slow  flight, 
and  stall  and  landings.  The  cross-country  flight  included  additional  grades 
on:  (a)  Flight  planning  and  filing,  (b)  VOR  orientation  and  tracking,  (c) 

simulated  loss  of  horizon,  (d)  change'  in  flight  plan,  and  (e)  landings  at 
several  airports  (if  feasible). 

"In  general,  the  criteria  for  "average"  was  that  established  by  the  FAA 
Private  Pilot  Flight  Test  Guide  AC  61  (il).  Individual  grades  were  assigned 
on  observed  performance  in  three  areas;  and  an  overall  grade  was  recorded. 

The  graded  areas  were: 

1.  Procedure,  retention,  and  recall.  The  subject  was  expected  to  be 
knowledgeable  concerning  FAR,  Part  61  - Certification:  Pilots  and  Flight 
Instructors,  and  Part  91  - General  Operations  and  Flight  Rules.  Written 
quizzes  were  administered  to  each  subject  prior  to  the  first  two  flights,  tut 
evaluators  were  expected  to  ask  questions  and  observe  the  subject's  adherence 
to  specific  rules  and  procedu.es  as  required  for  safety  of  flight. 

2.  Judgment  and  problem  solving.  Grades  in  this  area  were  based  on  the 
subject's  ability  to  use  whatever  information  was  available  to  him  and  to 
apply  it  as  would  be  expected  for  his  levc’  of  pilot  certification.  Especial! 
important  was  the  subject's  judgment  and  actions  as  related  to  flight  safety. 

3.  Motor  coordination.  The  "average"  pilot  was  expected  to  demonstrate 
the  ability  to  maintain  the  aircraft  in  a safe  flight  attitude  under  ail 
normal  conditions.  For  all  maneuvers  it  was  required  that  airspeed  be 
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Figure  3.  Reproduction  of  United  Airlines  Flight  Crew 
Enroute  Proficiency  Check  form. 
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Figure  A.  Reproduction  c.f  United  Airlines  Pilot  Flight 
Training  Record,  (Continued  on  next  page. ) 
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Figure  5.  Reproduction  of  United  Airlines  Flight  Training 
Center  - Pilot  Flight  Training  Record 
(DC-8  Equipment).  (Continued  on  next  page.) 
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| maintained  within  +_  5 mph,  altitude  within  100  ft,  and  heading  within 

+_  10°.  In  addition,  the  subject  was  expected  to  be  able  to  quickly 
recognize  unsafe  flight  conditions  and  to  take  proper  action  when  needed. 

L 4.  Overall  grade.  This  concerned  the  evaluation  of  the  overall  skill 

and  knowledge  demonstrated  on  each  flight.  The  subject  was  given  comments 
on  his  performance , but  no  information  on  grades  or  the  rating  system. 

In  a simjlar  way  but  using  a more  sophisticated  format,  United  Airlines 
(UAl)  makes  a concerted  effort  to  use  "objective"  test  procedures  for 
assessing  pilot  proficiency.  The  grading  system  now  in  use  is  a pass/fail 
system,  and  the  evaluation  criteria  used  are  contained  in  the  airplane  flight 
manuals  which  were  established  under  the  "Specific  Behavioral  Objectives" 
system.  The  pilot  proficiency  rating  is  given  in  a more  general  way  in 
Figure  2.  ' t documents  how  the  pilot  has  been  trained  and  checked  and  that 

the  United  Airlines  flight  training  requirements  were  met  and  completed. 

Figure  3 is  a reproduction  of  a UAL  form  which  shows  the  systematic 
arrangement  of  crew  rating  requirements  in  an  operational  sequence  from  the 
flight  preparations  to  the  final  approach  and  landing  procedure  at  the  end  of 
a flight.  In  addition,  Box  6 on  that  form  contains  criteria  for  comments  Gn 
general  requirements  which  the  crew  member  must  meet  during  the  en  route 
proficiency  check.  There  also  is  space  for  remarks  and  recommendations 
concerning  shortcomings,  retraining,  and  fljght  or  crew  assignment. 

Figures  4 and  5 are  reproductions  of  UAL  forms  which  contain  very 
detailed  information  on  the  pilot's  record  for  simulator  and  inflight 
training  as  requested  by  the  company's  flight  training  center.  The  training 
record  and  grading  standards  are  given  in  a very  general  form  in  Figure  6. 

It  should  also  be  pointed  out  in  this  context  that  all  pilots-in- 
command  operating  FAA  aircraft  must  satisfactorily  complete  periodic 
proficiency  checks;  and  the  results  of  these  checks  are  recorded  on  FAA  Form 
4040-2  (see  Figure  7),  The  Record  of  Check  Flight  includes  12  categories 
containing  items  of  significance  to  the  safe  operating  and  piloting  an 
aircraft  as  shown  in  Figure  7 . The  check  pilot  will  mark  only  those  items 
that  are  applicable  to  a particular  check  ride  or  proficiency  test,  and  the 
grading  on  each  item  is  either  "satisfactory"  or  "unsatisfactory. " Compared 
; to  some  of  the  other  examples  given  in  this  report,  this  system  of  rating 

pilot  proficien< y is  rather  unsophisticated  and  does  not  lend  itself  to  a 
more  differentiated  assessment  of  performance. 

The  proficiency  ratings  of  British  airline  pilots  as  directed  by  the 
Civil  Aviation  Authority  (CAA)  is  similar  to  that  of  its  American  counterpart. 
Methods  of  assessment  generally  vary  with  the  individual  airlines,  and  most 
of  them  also  apply  a simple  pass/fail  criterion;  with  a few  requesting  a 
somewhat  expended  scale  providing  for  remarks  like  "very  good,"  "good," 
"satisfactory,"  and  "unsatisfactory,"  or  like  the  European  Division  of 
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FAA  Fwm  4040-2  11-7* 


Figure  7,  Reproduction  of  Pilot/Flight  Engineer/Navigator  Tlight 
Record  and  Record  of  Check  Flight  (FAA  Form  4-040-2). 
(Continued  on  next  page.) 
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Figure  7 (Continued) 


British  Airways,  "above  average,"  "average,"  etc.  Usually,  the  biannual 
competency  cheeks  arc  treated  as  refresher  training  as  well,  and  the  rating 
is  of  the  pass/fail  type  with  most  of  the  pilots  passing  this  affair.  The 
judgment  of  the  inspector  is,  of  course,  subjective  and  the  CAA  does  not 
require  or  specify  detailed  evaluation  criteria.  However,  certain  ground 
rules  and  standards  are  available  in  the  "Notes  for  the  Guidance  of 
Authorized  Instrument  Rating  Examiners"  published  by  the  CAA  in  London  (CAP 
170),  since  it  is  normal  practice  to  combine  the  instrument  rating  and 
competency  check.  A combined  instrument  rating  and  competency  check  form  is 
available  for  this  purpose;  and  all  items  annotated  on  that  form  as  being 
relevant  to  the  instrument  rating  renewal  must  be  rated  at  least  "satisfactory" 
in  order  to  pass  the  proficiency  check. 

The  German  Lufthansa  has  outdone  the  German  Air  Force  in  developing  a 
"Pilot's  Proficiency  Report"  which  permits  a rater  to  specify  in  great  detail 
pilot  performance  during  the  training  and  overall  proficiency  assessment 
procedure.  The  report  form  (Figure  8)  contains  five  main  areas  of 
competence,  which  describe  distinct  and  observable  modes  of  behavior 
(criteria).  By  using  a numerical  grading  system  from  1 to  5 (1  indicating 
"unusually  effective,"  5 indicating  "unsatisfactory"),  the  instructor  or 
flight  inspector  may  rate  the  pilot  in  regard  to  the  required  level  of 
performance.  But  the  system  is  even  more  differentiated  in  that  the  grades 
2,  3,  and  8-  are  subdivided,  so  that  actually  9 levels  of  competence  are 
available  to  choose  from.  Moreover,  the  five  main  areas  contain  the  following 
items: 

1.  Knowledge  (Knowledge  of  Flight  Rules,  Regulations,  and  Mechanical 
Principles) . 

Criteria:  Is  familiar  with  aircraft  performance  characteristics; 
can  explain  aircraft  systems  and  knows  their  locations  and 
limitations;  understands  the  technical  relationships  of  aircraft 
systems  and  their  normal  operations;  is  familiar  with  emergency 
procedures;  knows  the  operational  rules  and  flight  procedures. 

2.  Use  of  Checklist  (Philosophy  and  Application). 

Criteria:  Uses  the  checklist  conscientiously  arid  conducts  all 
necessary  control  actions  in  a systematic  and  timely  fashion. 

3.  Flying  Ability. 

3.1  Aircraft  Handling  (Use  of  Controls) 

Criteria : Controls  the  aircraft  with  sensible  and  good 
coordination;  does  not  overcontrol  during  corrections; 
demonstrates  steadiness  in  the  control  actions. 
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Figure  8.  Reproduction  of  German  Lufthansa  Pilot's  Proficiency  Report 

Transition  Training  (front  and  back  pages).  (Continued  on  next 


3.2  Basic  Flying  (Integration  of  Flight  Procedures) 


Criteria : Maintains  orientation  and  position  in  space ; reads 
instruments  correctly  and  corrects  unwanted  deviations: 
intermittently  scans  airspace;  anticipates  changes  in  flight 
conditions;  maintains  course  and  desired  flight  path;  keeps 
systems  within  tolerances. 

3.3  Takeoff  and  Climh-out  (Execution  of  Prescribed  Maneuvers) 

Criteria : Executes  normal  procedures  under  various  conditions 
(weight , crosswind,  flap  position,  noise  abatement);  when  required, 
aborts  takeoff  in  time  and  safely  stops  aircraft;  compensates 
for  engine  failure  after  Vp  and  proceeds  in  accordance  with 
regulations;  stays  within  flap  speed  schedule. 

3.4  Instrument  Approaches  (Landing  Approaches  Under  IFR  Conditions) 

Criteria : Knows  alJ  relevant  subjects  and  conducts  appropriate 
briefings;  files  in  accordance  with  the  approved  procedures  and 
observes  ATC  clearance;  proceeds  in  a timely  manner  considering 
all  available  information;  stabilizes  flight  conditions  and  stays 
in  slot;  transitions  well  from  ILK  to  VFR;  decides  to  abort 
approach  and  to  go  around,  if  indicated. 

3.5  Visual  Approaches  (Landing  Approaches  Under  VFR  Conditions) 

Criteria:  Observes  the  various  VFR  landing  procedures  (normal, 
low  circling,  different  flap  settings);  accurately  determines 
downwind  and  base-leg  approach  under  the  prevailing  flight 
conditions  and  configuration  for  proper  line-up  in  slot;  makes 
glidepath  and  centerline  corrections  and  stabilizes  the  aircraft 
relative  to  touchdown  area;  decides  to  abort  approach  and  to  go 
around,  if  indicated. 

3.6  Landing  (Execution  of  Landing  the  Aircraft  After  IFR  or  VLR 
Approach  Including  Touchdown  Procedure  or  Co  Around) 

Criteria:  Initiates  flare  at  the  appropriate  time;  touches  down 
on  centerline  and  within  touchdown  area;  observes  after-touchdown 
procedures;  lands  aircraft  unoer  unfavorable  conditions  (crosswind, 
darkness,  unusual  configurations);  initiates  go  around  at  the 
right  time  (attitude,  power)  3nd  takes  timely  and  adequate  actions 
to  land  the  aircraft. 


4.  Abnormal  and  Emergency  Procedures  (In  Accordance  With  Flight 
Manuals  and  Crew  Participation) . 

Criteria:  Recognises  kind  and  amount  of  system  failures;  takes 
appropriate  and  immediate  action;  uses  Abnormal  and  emergency  Proce- 
dure List  in  a timely  and  coordinated  way;  keeps  aircraft  under 
control . 

5.  Professional  Ability  (Abilities  and  Behavior  Important  to  the 
Pilot's  laslcTi 

Criteria:  knows  how  to  combine  instructional  advice  and  personal 
experience ; recognizes  situations  which  demand  decisions  and  takes 
timely  and  appropriate  actions;  establishes  the  right  priorities; 
acts  calm  and  controlled;  performs  effectively  under  stress. 

There  is  additional  space  left  below  each  of  the  competence  areas  to 
supplement  remarks  about  the  behavior  of  the  candidate  or  about  special 
features  of  his  performance  which  deserves  attention;  and  such  statements  can 
be  expanded  on  the  last  page  of  the  performance  report  form  under  "Comments 
and  Recommendations"  (see  Figure  8). 

The  total  form,  including  the  observations,  grades,  and  recommendations, 
is  shown  to  the  trainee  or  rated  individual  at  the  end  of  the  procedure;  and 
the  rated  person  has  the  right  to  a written  reply  or  rebuttal  in  case  of 
disagreement.  There  is  also  an  attempt  made  by  Lufthansa  to  provide  the 
instructor  or  rater  with  a kind  of  standardized  rating  procedure. 


Figure  9.  Normal  distribution  curve. 
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The  Lufthansa  rater  is  advised  to  use,  if  possible,  the  normal  distribu- 
tion curve  as  the  basis  of  his  grade  assignments  (see  Tigure  9).  In  this 
process,  he  should  determine  if  (i)  a rating  within  a certain  area  and  on  a 
specific  criterion  is  indicated,  (ii)  the  grade  3 is  an  adequate  rating,  (iii) 
a grade  2 or  4 would  be  more  appropriate,  or  (iv)  a grade  1 or  3 can  be 
justified.  A satisfactory  performance  is  mandatory  either  as  a measure  of 
normal  progress  during  transition  training  or  as  an  accepted  standard  of 
pilot  performance.  A flight  training  test  or  a proficiency  check  is 
considered  as  passed,  if  all  graded  criteria  are  rated  as  at  least  standard 
performance. 

In  the  United  States,  the  FAA  is  aware  of  substantial  variations  in  the 
manner  in  which  inflight  performance  is  assessed,  and  in  the  reports  which 
reflect  the  evaluation,  judgment,  ratings,  and  results  of  the  flight  tests 
conducted  by  FAA  examiners.  The  official  performance  guidelines,  descriptive 
and  detailed  as  they  are,  do  not  presently  provide  for  a real  objective 
assessment  of  the  procedures,  maneuvers,  and  operations,  and  even  less  for  the 
behavioral  characteristics,  abilities,  and  skill  of  the  applicant  or  pilot  to 
be  tested.  As  a remedy,  the  FAA  is  conducting  seminars,  training  courses, 
and  workshops  for  inspectors  and  examiners.  Within  the  present  system,  this 
will  help  to  Increase  the  reliability,  accuracy,  and  consistency  of  the 
subjective  ratings. 

By  and  large,  it  can  be  seated  that  there  are  many  subjective  systems 
available  and  in  use  which  have  been  proven  practical  and  efficient  for 
assessing  pilot  performance.  They  can  be  adapted  to  any  operational 
situation,  expanded  to  provide  needed  or  desired  information,  and  kept  on 
record  during  the  professional  life  of  a pilot.  Although  the  dynamics  of  the 
flight  environment,  the  complexity  of  the  phenomena  to  be  observed,  and  the 
speed  with  which  they  occur  impose  a heavy  burden  on  the  examiner,  quantita- 
tive rating  scales  for  the  manual  recording  and  grading  of  procedures  are 
still  very  popular  with  the  airlines  and  official  organizations.  They  permit 
the  examiner  to  evaluate  those  qualitative  behaviors  reflecting  on  the 
examinee's  ability  to  cope  effectively  and  safely  with  the  various  demands, 
requirements,  and  potential  hazards  of  the  total  flight  environment. 

Objective  Measurements  Using  Flight  Simulators . 

A.  Fixed-Wing  Aircraft.  There  have  been  several  past  efforts  under- 
taken to  design,  dc'/clop,  arid  use  simulator  systems  for  objectively  measuring 
pilot  performance  (9,10,11,12).  For  example,  part-mission  simulation 
performance  measures  were  aimed  at  the  landing  procedure,  statistically  the 
source  of  most  aircraft,  accidents.  In  the  course  of  various  studies, 
starting  with  a comparison  of  center  sticks  versus  side  control  sticks  in 
1970,  the  Crew  Station  Design  Facility  at  the  Aeronautical  Systems  Division, 
Wright-Fattr^on  A(-‘3,  Ohio,  had  a need  for  an  objective  and  quantitative 
method  of  evaluating  pilot  performance  during  Instrument  Landing  Systems  (ILS) 
approaches  and  landings.  To  meet  this  requirement,  a numerical  scoring 
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system  was  designed  and  tested  which  yielded  relatively  consistent  and 
reliable  measures  of  landing  performance  (22).  In  various  studies  and 
comparisons  with  other  measures,  it  demonstrated  its  usefulness  to  the 
intended  purpose. 

In  1971,  Hill  and  Goebel  (19)  developed  automatic  measures  of  pilot 
performance  for  a General  Aviation  Trainer  (GAT-1).  Two  years  later,  they 
expanded  their  investigation  through  a re-analysis  of  their  earlier 
statistics  and  the  addition  of  a compensatory  tracking  task.  The  approach  was 
based  on  two  separate  experiments  carried  out  by  using  the  GAT-1:  A basic 
experiment  with  326  measurements  on  each  of  30  subjects  in  three  different 
experience  groups,  and  an  expanded  experiment  with  2,43 6 measurements  on  each 
of  30  subjects  front  the  same  three  groups.  The  first  experiment  included  four 
different  flight  tasks  lasting  about  10  minutes  each;  the  second  experiment 
consisted  of  these  and  six  additional  tasks  (18). 

The  results  of  the  experiments  showed  that  there  is  little  difficulty  in 
obtaining  measurements  that  correlate  with  experience.  Tables  of  more  than 
400  important  data  elements  were  prepared  by  the  authors  with  group  means,, 
standard  deviations,  and  further  cross- tabulations  that  showed  which  tasks 
and  measurements  were  best  at  discriminating  among  pilots.  The  outcome  of 
the  study  also  indicated  that  the  statistical  approach  used  by  Hill  am 
Eddowes  (19)  was  not  effective  for  the  development  of  a practical  pilot 
performance  measurement  system;  and  that  different  procedures,  equipment,  and 
means  had  to  be  used  to  achieve  the  intended  goal. 

Shipley,  Gerlach,  and  Brecke  (32)  recorded,  analyzed,  and  discussed  the 
data  obtained  from  student  pilots  while  flying  a T 4-C  simulator.  Two  some- 
what different  methods  of  collecting  data  were  considered.  The  first  one  was 
the  use  of  a checklist  by  an  expert  observer.  The  observations  could  have 
been  made  during  the  subject's  actual  performance  or  they  could  have  been  made 
by  inspecting  a video-recording  sometime  after  their  performance.  The  second 
method  considered,  and  ultimately  adopted,  was  the  use  of  an  electronic 
analog-to-digita.l  recording  device  to  record  the  several  electrical  impulses 
emanating  from  and/or  entering  the  simulator's  control  and  instrumentation 
systems.  A ten-channel,  recording  device  was  used  to  obtain  information 
about  flight  instruments,  such  as  altimeter,  airspeed,  rate  of  descent, 
heading,  attitude,  power,  and  throttle  activation. 

The  records  were  coded,  transferred  to  tape,  and  treated  to  indicate 
experimental  details.  The  tapes  were  then  evaluated  using  a three  criterion 
scoring  procedure;  namely,  time  on  "target,"  bit  rate,  and  error  amplitude. 
Summary  scores  of  the  performance  of  each  subject  were  computed  and  subjected 
to  two  different  analyses  of  variance  to  test  for  differences  in  performance. 
Single  observation  of  response  time  and  maximum  altitude  for  each  trial  were 
also  analyzed.  The  graphic  performance  plots  revealed  significant  group 
differences,  among  other  things. 


26 


Four  sets  of  graphic  representations  of  the  data  were  used  as  an  alterna- 
tive for  judging  the  validity  of  the  output  of  the  statistical  computations. 

One  result  of  the  program  for  generating  the  graphic  displays  was  the  discovery 
of  two  easily  observed  and  computed  measures  of  performance  quality,  namely, 
performance  time  or  time  on  target  and  maximum  altitude  of  the  vertical  S-A 
maneuver.  (The  Vertical  S-A  consisted  of  a series  of  alternating  climbs  and 
descents  flown  at  a constant  rate  of  speed  (1,000  ft/min)  and  heading.)  These 
two  measures  were  potentially  useful  as  estimators  of  general  differences  in 
performance  in  subsequent  research. 

Another  study  was  recently  conducted  by  Carter  (5,6),  who  used  the 
Northrop  LAS/WAVS  air  combat  simulator  for  automated  performance  measurements 
(APM).  He  identified  a set  of  measures  for  the  evaluation  of  air-to-air 
contiat  tactics  and  various  statistical  techniques  adequate  for  this  process. 

The  effort  consisted  of  nine  major  different  tasks;  namely,  maneuver  selec- 
tion, development  of  appropriate  and  valid  evaluation  methods,  measure 
analysis,  measure  definition,  software  development,  data  collection,  data 
reduction,  and  measure  selection. 

The  maneuvers  selected  for  the  APM  study  were  the  barrel  roll  attack,  the 
high  yo-yo,  and  the  lag  roll.  While  data  were  initially  collected  on  all 
three  maneuvers,  problems  with  the  autopilot  bogey  on  the  latter  two 
maneuvers  resulted  In  a subsequent  decision  to  limit  the  study  to  the  barrel 
roll  attack. 


Highly  detailed  behavioral  objectives  were  developed  for  each  of  the 
maneuvers  contained  in  the  introductory  phase  of  the  Navy  F43  RAG  syllabus 
(14).  The  methodology  and  results  of  this  task  are  documented  in  Carter  (5), 
The  detailed  understanding  of  air  combat  maneuvers  gained  in  this  task 
provided  an  important  basis  for  all  subsequent  tasks  in  the  APM  study. 


Special  scoring  forms  were  developed  to  provide  a much  more  detailed  and 
systematic  instructor  evaluation  of  student  performance  than  the  grading 
techniques  normally  used  in  flight  training.  The  approach  of  this  problem  was 
based  on  the  critical  incident  technique  originally  developed  by  John  Flanagan 
in  the  1950's  (for  a short  description  of  this  technique,  cf.  14).  The  rating 
form  was  designed  to  record  instructor  observations  and  judgments  relating  to 
the  following  in  each  run:  (i)  critical  errors  occuring  during  the  run;  (j.i) 
the  qualitative  value  of  critical  parameters  at  each  of  several  points  during 
the  run;  (iii)  the  quality  of  the  end-position  achieved;  and  (iv)  an  overall 
grade  for  the  run.  These  data  were  ultimately  reduced  to  punched  cards  by 
assigning  numerical  values  to  the  judgment  categories  in  the  qualitative 
scales  developed  for  use  with  the  form. 

Seven  FAO  student  pilots  and  six  F4J  instructor  pilots  flew  16  barrel  roll 
attacks  against  an  autopilot-controlled  bogey,  for  a total  of  208  simulator 
runs.  A total  of  552  objective  performance  measures  and  an  average  of  35 
subjective  performance  measures  were  obtained  on  each  run.  Using  the 
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simulator's  replay  capability,  64  of  the  original  203  runs  were  evaluted 
independently  by  three  different  instructors  to  obtain  estimates  of  inter- 
observer  reliability.  Sixteen  of  these  64  runs  were  evaluated  a second  time 
by  the  same  three  instructors  to  obtain  estimates  of  intra-observer 
reliability.  A master  tape  was  constructed  which  contained  all  of  the 
subjective  and  objective  measures  obtained  for  each  run  in  a format  that 
permitted  statistical  analyses  of  any  desired  subset  of  subjects,  evaluators, 
or  performance  variables.  Several  different  univariate  and  multivariate 
analyses  were  performed  on  selected  subsets  of  the  data. 

In  general,  results  of  the  measure  selection  analysis  yielded  several 
objective  measures  which  were  used  to  augment  and  facilitate  instructor 
evaluation  and  diagnosis  in  introductory  air-to-air  tactics  (6).  Several  sets 
of  automated  measures  were  identified  which  had  high-multiple  correlations 
with  both  instructor  judgments  and  value  of  critical  objective  parameters  at 
later  points  in  the  maneuver. 

B.  Rotary  Wing  Simulator.  VreuI  and  Obermayer  (37)  studied  helicopter 
crew  performance  through  the  analysis  of  12  maneuvers  in  a "Oayeopter."  This 
effort  consisted  of  time  history  measures  (e.g.,  time  on  target,  time  out  of 
tolerance),  amplitude  distribution  measures  (e.g,,  mean  and  median  values  of 
the  control  movement  deviation),  and  frequency  domain  measures,  which 
included  such  things  as  autocorrelation  functions,  power  spectral  density 
functions,  and  transfer  model  parameters.  Their  interest  rested  more  with  the 
mathematics  and  modeling  techniques  for  total  system  response  than  with  the 
human  factors  involved.  Vreul  and  Obermayer  concluded  that  the  engineering 
hardware  and  the  behavioral  research  methods  are  available  to  provide 
objective  pilot/system  performance  measurements  of  sufficient  accuracy.  The 
major  constraints  appeared  to  be  primarily  related  to  the  amount  of  time  and 
effort  required  to  define  the  parameters  and  to  test  the  validity  of  the 
method  and  results,  but  data  collection  and  handling  are  easily  accomplished 
by  computers  and  automatic  data  processing  (ADP).  In  order  to  reduce  the 
costs  of  obtaining  performance  information  and  to  maximize  their  utility  or 
applicability,  the  authors  suggested  that  methods  and  software  should  be 
improved. 

Specifically,  the  cost  of  empirical  data  collection  for  obtaining 
quantifiable  information  on  performance  parameters  can.  be  reduced  if:  (i) 
attempts  are  made  to  collect  only  the  type  of  results  which  can  bo  generalized, 
and  (ii)  only  such  information  is  collected  that  can  be  standardized  and 
catalogued  for  use  by  others. 

The  data  collected  by  Vreul  and  Obermayer  (37)  meet  these  criteria.  They 
discriminate  very  well  among  their  selected  parameters.  In  addition,  the 
authors  made  some  measurements  in  actual  flight. 
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Table  2.  T-37  Flight  Variables  Recorded  by  Knoop  (26) 


Variable 

Units 

Samples 
per  Second 

Airspeed 

knots 

100 

Pitch 

deg. 

100 

Roll 

deg. 

100 

Stick  Position  (Long.) 

deg. 

100 

Stick  Position  (Lat.) 

deg. 

100 

Rudder  Position 

deg. 

100 

Heading 

deg. 

10 

Altitude 

feet 

10 

Vertical  Acceleration 

9's 

10 

Pitch  Rate 

deg./sec. 

10 

Roll  Rate 

deg.  /sec. 

10 

Yaw  Rate 

deg. /sec. 

10 

RPM  (both  engines) 

percent 

10 

Throttle  Positions 

deg. 

*10 

Flap  Position 

percent 

10 

Landing  Gear 

discrete 

10 

Speed  Brakes 

discrete 

IU 

Thrust  Attenuator 

discrete 

10 

Trim  Tab  Movements 

discrete 

10 

Time 

hrs./min./sec. 

10 

Record  Number 

integer 

10 
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1 Maneuver  entry 

2 45°  turn  point 
rjt  80°  turn  point 

4 1 35a  turn  point 

5 180°  turn  point  (midpoint  of  maneuver) 

6 1 35c  turn  point  (direction  opposite  from  0.1  -0.5) 

7 90°  turn  point 

8 45°  turn  point 

9 Maneuver  termination  (straight  and  level  flight) 


Figure  10.  Lazy  8 maneuver  profile. 
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Inflight  Performance  Measurements. 

A.  Fixed-Wing  Aircraft.  Extensive  inflight  research  in  fixed-wing 
aircraft  has  been  conducted  by  Knoop  and  Welda  (26)  and  Knoop  (25)  in  order 
to  develop  an  objective  performance  measuring  system  for  use  in  Undergraduate 
Pilot  Training  (UPT)  in  the  U.S.  Air  Force  (USAF).  This  was  accomplished  by 
an  automated  performance  measurment  system  which  was  reliable,  sensitive,  and 
accurate.  A T-37B  was  instrumented  to  record  the  flight  variables  listed  in 
Table  2. 


This  effort  was  at  first  directed  to  investigate  the  feasibility  of  using 
quantitative  measurement  techniques  for  two  of  the  flight  maneuvers  taught  in 
the  USAF  UP1  flight  syllabus,  namely,  the  Lazy  8 and  the  barrel  roll.  The 
Lazy  8 is  a maneuver  requiring  simultaneous  turning  and  climbing  or  descending 
in  such  a fashion  that  a regular  horizontal  figure  8 is  described  about  a 
selected  point  of  reference  located  on  the  horizon.  Figure  10  illustrates  the 
nine  maneuver  elements  of  the  Lazy  8.  The  element  numbers  coincide  with  the 
circled  task  analysis  number  used.  The  barrel  roll  consists  of  an  aerobatic 
roll  maneuver  of  360°  bank  about  a selected  reference  point  located  ahead  of 
the  aircraft.  The  sensors  and  recording  equipment  were  strictly  off-the-shelf 
components  that  had  proved  to  be  reliable  in  previous  flight  test  projects. 

An  extensive  computer  software  system  was  developed  with  which  to  reduce, 
calibrate,  and  analyze  the  recorded  data  from  the  Lazy  8 and  barrel  roll 
maneuvers,  and  to  compute  performance  measures.  Criterion  values  for  the  two 
maneuvers  were  developed  by  utilizing  task  analysis  data,  narrative 
descriptions,  and  recorded  inflight  maneu'/er  performance  of  a highly  qualified 
Air  Traininq  Command  instructor  pilot. 

The  data  were  systematically  sampled,  digitally  encoded  and  recorded 
on  magnetic  tape.  The  calibrated  records  were  then  inspected  to  produce 
printouts,  plots,  and  card  copies  of  selected  parameters  for  use  in 
the  data  analysis  procedure.  Typical  plots  for  the  Lazy  8 and  barrel  roll 
are  shown  in  Figures  11  and  12.  By  utilizing  the  recorded  data  obtained 
from  16  students  and  4 instructors,  experimental  performance  measures  were 
derived  through  an  iterative  analytical  approach. 

Study  results  indicated  that  Lazy  8 performance  assessment  can  be 
accomplished  using  the  flight  parameters  of  roll  angle,  pitch  angle,  and 
airspeed  in  a single,  summary  error  measure.  Barrel  roll  measurement  is 
dependent  upon  roll  and  pitch  angle,  acceleration,  and  roil  rate.  A definite 
relationship  between  roll  and  pitch  was  critical  to  the  measurements. 

In  a later  report  concerning  the  development  of  standardized  techniques 
for  deriving  and  validating  measures  of  operator  performance,  Connelly, 

Bourne,  Loental,  and  k'noop  (9)  described  the  theory,  structure,  and  implemen- 
tation of  a processor  (written  in  FORTRAN  IV)  that  can  accept  data  representing 
various  levels  of  operator's  skill  and  analyze  performance  measures  and 
validation  test  results.  The  theoretical  concept  of  their  study  and  the 
computational  techniques  were  thought  to  have  great  potential  for  this  type 
of  activity. 
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Figure  11.  Lazy  8:  roll/pitch,  airspeed/ time,  and 
altitude/time  plots. 
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Figure  13.  Flow  diagram  of  processor. 
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Table  3.  Possible  Criterion  and  Performance  Measure  Factors  Cited 

Connelly  et  al,  (8) 
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The  same  processor  was  used  for  measurement  problems  associated  with  five 
DPT  contact  training  maneuvers  flown  in  the  T - 37  aircraft,  namely,  barrel  roll, 
Lazy  8,  Clover  Leaf,  Split  S,  and  a normal  landing  (8).  The  activities 
necessary  for  obtaining  the  desired  measurements  included  several  steps,  such 
as  the  development  of  criteria,  the  determination  of  the  significance  of 
deviation  from  these  criteria,  the  search  for  candidate  performance  measures 
and  their  A OP  transformation,  their  validation,  and  the  design  of  an  adequate 
data  management  process.  A generalized  flow  diagram  of  the  process  is  given 
in  figure  13.  Some  possible  criterion  and  performance  measure  factors 
applied  in  this  context  are  shown  in  Table  3.  The  analytical  method  included 
the  identification  of  two  types  of  function  segments  (locus  and  sequence) 
within  a given  control  task,  wherein  the  set  of  dominant  measurement 
variables  is  consistent.  In  this  way,  portions  of  each  individual  task  and 
portions  of  each  task  segment,  in  which  the  operator's  primary  control 
functions  remained  consistent,  were  identified.  This  suggested  that  the 
specific  nature  of  the  continuous  or  discrete  measures  was  compatible  with 
the  intended  performance  assessment. 

B.  Rotary  Wing  Aircraft.  Billing  (1),  Billings,  Eggspuehler,  Gerke, 
and  Chase  (2),  and  Billings,  Gerke,  Chase,  and  Lggspuehler  (3)  delineated  a 
quantitative  and  objective  method  of  evaluating  pilot  performance  in  a 
Hiller  12-F  helicopter.  The  aircraft  was  instrumented  for  recording  rotor 
velocity  (rpm),  cyclic  and  collective  pitch  control  movements,  and  throttle 
position.  After  many  tryouts  and  calibration,  these  parameters  were  found 
promising  to  measure  pilot  performance  during  low- level  flights  of  varying 
demands  and  amounts  of  work,  in  particular  during  power  line  inspection^. 
Several  years  later,  the  authors  validated  their  previous  results  by 
conducting  experiments  with  a mixed  group  of  flight  instructors  and 
students,  recording  the  student's  electrocardiograms  as  indexes  of  workload 
and  fatigue.  The  findings  from  this  study  supported  their  hypothesis  that 
rotor  rotations  per  minute  (in  terms  of  rpm  variability)  was  a valid  index  of 
pilot  skill  in  helicopter  flight,  and  that  methods  used  in  these  experiments 
are  useful  tools  for  assessing  pilot  performance. 

Investigations  hy  personnel  of  the  U.S.  Army  Aeromedica • Research 
Laboratory  in  Fort  Rucker,  Alabama,  during  the  1974/77  time  period  concerned 
pilot  performance  during  nap-of-the-earth  (NOF),  low-level,  and  local  area 
flights  (13,24,29).  Most  of  the  experiments  wore  centered  about  tpe  assess- 
ment of  helicopter  crew  performance,  the  nature  of  the  flight  and  combat 
environment,  the  operational  demands,  perceptual  problems,  and  the  develop- 
ment of  . propriate  methods  of  workload  measurements.  Inflight  measurements 
of  the  aviator  and  the  recording  of  aircraFt  parameters  provided  results 
which  were  sensitive  to  workload  and  fatigue  by  extended  flight  durations. 

Performance  data  were  obtained  through  the  use  of  the  helicopter  inflight 
monitoring  system  (HIMS).  This  research  tool  provided  for  the  real  acquisi- 
tion of  all  major  aircraft  motion  and  pilot  control  parameters.  It  monitors 
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Table  4.  Helicopter  Flight  Parameters  Measured  and  Derived 
by  Kimball  et  al.  (24) 


Parameters  Measured 


Derived  Measures 


Pitch 

Roll 

Heading 
Position  x 
Position  y 

Acceleration  x 
Acceleration  y 
Acceleration  z 
Roll  Rate 
Pitch  Rate 
Yaw  Rate 
Radar  Altitude 

Barometric  Altitude 
Airspeed 
Flight  Time 
Rotor  Rr’M 
Throttle 

Cyclic  Stick  (Fore-Aft) 
Cyclic  Stick  (Left  Right) 
Collective 
Pedals 


Pitch  Rate 
Roll  Rate 
Rate  of  turn 

Constant  Eiror,  Average  Absolute  Error,  RMS  Error 
Ground  Speed,  Constant  Error  Average  Absolute  Error, 
RMS  Error 


Roll  Acceleration 
Pitch  Acceleration 
Yaw  Acceleration 

Rate  of  Climb,  Average  Absolute  Error.,  Constant  Error.. 

RMS  Error 
Rate  of  Climb 


Control  Position,  Absolute  Contro!  Movement  Magnitude, 
Positive  Control  Movement  Magnitude,  Negative  Control 
Moverm  it  Magnitude,  Absolute  Average  Control  Movement 
Rate.  Av,  age  Positive  Control  Movement.  Rate,  Average 
Negative  Control  Movement  R ue,  Control  Hcersals, 
Instantaneous  Control  Reversals,  Control  Steady  State, 
Control  Movement 


and  records  aircraft  motion  in  all  six  degrees  of  freedom  as  well  as  all 
pilot  control  movements.  A list  of  the  parameters  measured  and  derived  is 
shown  in  Table  4. 

The  helicopter  pilot  performance  measurements  were  supported  by 
industry  developments  in  the  area  of  pilot  contribution  to  aircraft  system 
operation.  An  example  of  this  effort  is  a technique  to  gather  empirical  data 
on  the  inflight  acquisition  of  task  sequences  and  task  times  designed  by  the 
Vought  Corporation  in  Dallas,  Texas.  Vought  had  demonstrated  key  features 
of  the  proposed  system,  using  existing  equipment,  in  a recent  helicopter 
vision  study  contracted  by  the  U.S.  Army  Aviation  Systems  Command.  The 
visual/audio  data  can  be  supplemented,  complemented,  or  verified  with  other 
system  measures  which  are  common  to  the  instrumentation  of  all  new  military 
aircraft.  These  include:  stick/rudder/ throttle  positions,  rates  of  deflec- 
tion, and  forces;  aircraft  flight  profile;  aircraft  subsystems  moding  and 
performance  (15). 

The  measures  are  available  to  record  what  the  pilot  is  doing  to  operate 
the  aircraft  within  prescribed  mission  tolerance  and  how  the  aircraft  is 
responding.  Such  data,  when  reduced  and  processed,  as  in  the  Vought  Human 
Performance  model,  provide  graphic/numeric  readout  of  accuracy  of  performance 
to  prespecified  tolerances. 

Advanced  Inflight  Monitoring  Systems. 

In  retrospect,  the  concept  of  an  automatically  recording  inflight 
monitoring  system  for  air  transport  type  aircraft  emaudlcd  as  a means  to 
increase  flight  safety.  As  Ferrarese  (1?)  pointed  out,  there  exists  a 
credibility  gap  when  pilots  repori  that  any  given  flight  is  operated  in 
accordance  with  established  procedures,  that  the  aircraft’s  systems  function 
normally,  and  that  there  are  no  safety  problems  on  the  ground  and  in  the  air. 
System  malfunctions,  deviations  from  accepted  practice,  and  pilot  errors  do 
occur.  The  causative  factors,  such  as  internal  conditions  and  environmental 
forces  having  adverse  safety  effects,  .-re  most  difficult  to  identify  and  it 
is  sometimes  impossible  tc  assess  their  impact  from  the  cockpit. 

The  means  to  dost  this  Inflight  information  gap  is  found  in  new, 
technically  advanced  flight  recorder  equipment,  iloderr  logic  systems  and 
mathematical  nvdols  can  be  employed  to  gather  Information  concerning  the 
performance  of  the  aircraft  and  of  the  pilots;  and  means  arc  available  to 
reduce  such  information  into  some  understandable  end  useful  torn:.  High-speed 
analysis  systems  can  compare  the  obtained  inionrution  to  established  norms. 

In  order  to  measure  and  evaluate  performance,  one  must  compare  ’’what  should 
happen"  to  "what  is  actually  happening."  Flight  recording  and  analyzing 
systems  which  can  do  this  are  a technical  reality. 

As  to  the  possible  use  of  automatic  inflight  recording  for  obtaining 
proficiency  measures,  Ferrarese  U2)  stated: 
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"A  good  example  might  be  the  practice  of  reaffirming  pilot 
competence  with  respect  to  flying  the  instrument  landing  system 
(IL5)  each  .six  months.  Is  this  really  necessary  if  during  actual 
operation  the  ILS  flight  is  always  conducted  within  the  safe-flight 
envelope,  and  this  is  a matter  of  record?  The  system  can  identify 
those  who  do  veil.  It  is  thereby  '■ossible  for  the  individual  and 
the  operator  to  be  relieved  from  certain  portions  of  aircraft 
flight  checks  at  fixed  intervals.  Likewise,  those  who  depart  from 
established  norms  because  of  proficiency  problems  may  be  given 
training  as  the  situation  dictates,  rather  than  at  some  fixed  period. 

"in  a typical  system,  safe- flight  envelopes  or  programmed 
operating  limits  are  described.  Mathematical  models  of  these 
envelopes  or  norms  arc  programmed  in  computers.  Flight  data  are 
fed  into  the  computers  and  compared  to  the  stored  models.  All 
excursions  are  identified  and,  where  appropriate,  given  further 
qualitative  and  quantitative  analysis.  Part  of  the  analysis  will 
be  to  determine  if  the  stored  model  is  valid  or  in  need  of  change, 
whether  the  variables  are  properly  considered,  and  if  the 
airborne  data  are  adequate,  as  well  as  determine  the  adequacy  of 
procedures,  equipment  and  techniques.  This  operation  is  a most 
critical  part  of  the  system  and  requires  input  from  all  elements 
of  the  industry.  Flight  crews,  engineers,  medics,  supervisors, 
ground  personnel  may  ail  be  brought  into  the  picture." 

Airline  management  has  had  a long  standing  interest  in  the  improvement  of 
proficiency  assessment  of  airline  pilots.  Current  sampling  of  a -pilot's 
performance  consists  of  one  line  check  and  two  proficiency  checks  per  year. 

A line  check  is  an  audit  of  pilot  performance  durinn  a flight  over  a typical 
part  of  the  route  served  by  the  airline,  and  it  is  normally  made  by  an 
airline  check  pilot  or  an  F/ inspector.  Several  major  airlines  use  the 
flight  simulator  extensively  for  training  and  proficiency  checks  of  their 
pilots,  The  simulator  can  be  equipped  with  the  necessary  devices  to  obtain 
not  only  an  aircraft  type  rating,  but  also  for  evaluating  the  adequacy  of  the 
pilot's  line  performance,  if  the  performance  is  measured  against  professional 
flying  standards  on  an  adequate  and  factual  basis.  The  question  must  now  be 
asked  whether  such  techniques  can  also  be  used  under  actual  flying  conditions. 

Indeed,  flight  monitoring  and  analysis  systems  are  available  and  are  being 
used  to  assess  pilot  performance  in  objective  and  measurable  terms.  Such 
automated  performance  measurement  systems  inherently  permit  the  assessment  of 
pilot  performance  to  be  highly  sensitive,  valid  and  reliable,  since  perform- 
ance can  be  recorded  on-line  for  a large  number  of  system  variables.  Greater 
accuracy  regarding  the  performance  of  pilot  and  aircraft  under  the  prevailing 
flight  conditions  is  provided  by  an  automated  system  than  by  a human  observer, 
since  more  pilot  actions,  aircraft  responses,  and  flight  parameters  can  be 
recorded  within  a certain  period  of  time.  E3y  automatically  analyzing  the  data 
so  obtained,  a high  degree  of  objectivity  and  reliability  is  guaranteed  which 
cannot  possibly  be  afforded  by  human  observation. 
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Examples  of  these  systems  will  now  be  described  and  their  use  for  the 
measurement  of  pilot  performance  will  be  discussed.  The  selection  of  the  two 
systems  was  based  on  their  availability  at  the  time  this  report  was  prepared; 
and  it  is  not  inferred  that  there  are  no  other  systems  available  or  in  the 
design  stage,  which  could  not  be  applied  or  modified  for  the  purpo;  of 
automatically  recording,  analyzing,  and  measuring  pilot  or  aircrew  perform- 
ance. At  present,  the  two  systems  described  below  come  very  close  the 
concept  of  an  advanced  inflight  monitoring  system  as  envisioned  by  the  FAA. 

Concern  about  flight  safety  was  essential  for  American  Airlines  (AA)  to 
propose,  develop,  and  use  the  "Astrolog"  program  (30).  Based  on  operational 
experience,  several  desirable  attributes  of  a safe,  flight  operation  were 
described  in  words  and  then  converted  into  specific  numerical  limits.  This 
process  delineated  satisfactory  flying  performance  in  a workable  digital  form. 
The  three  parties  that  participated  in  the  process  of  deciding  on  what  the 
operational  envelopes  should  be  were  the  American  Airlines  piloting  manage- 
ment, the  Allied  Pilots  Association,  and  the  FAA.  In  setting  operational 
standards  concerning  the  size  of  the  various  envelopes  on  speed,  altitude, 
attitude,  etc.,  the  amount  of  deviation  from  those  standards  was  recorded  and 
analysed.  Automatic  data  handling  and  processing  techniques  were  extensively 
used  in  this  process.  The  software  could  be  adjusted  to  accommodate  new 
information  and  changing  l'equirements , 

The  "Aircraft  Integrated  Data  System"  was  installed  in  the  BAC-111 
aircraft  and  employed  for  the  intended  purpose  for  several  years.  In  order  to 
keep  the  amount  of  data  at  a manageable  minimum,  the  data  processing  method 
was  based  on  the  management-by-exccption  concept;  i.e.,  only  deviations  from 
the  "standards"  were  recorded,  and  a primary  document  known  as  an  "Exception 
Report"  was  rendered. 


Table  5.  American  Airlines  Astrolog  Exception  Report 


DATE  04  01  69  FLT  1014  LEG  1 ACFT014  CAPT  NO  12345 

TIME  FROM  200  FT  AFL  TO  TOUCHDOWN  32  SEC  CPI 
TIME  FROM  50  FT  AFL  TO  TOUCHDOWN  21  SEC  CRT 

FUEL  FLOW  VARIATION  BETWEEN  85  FT  AFL  AND  51  FT  AFL  2100  PPH  2345  GMT  CPT 
OUT  2230  GMT  OFF  2235  GMT  ON  2346  GMT  IN  2350  GMT 
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The  Exception  Reports  were  used  by  AA  supervisory  pilots  to  initiate 
corrective  action  appropriate  to  each  specific  situation.  A sample  of  an 
Astrolog  Exception  Report  is  shown  in  Table  5.  In  case  additional  information 
was  needed,  two  other  kinds  of  machine-produced  documents  were  available. 

They  would  provide  trend  information  by  exception  type.  When  widespread 
instances  of  a particular  deviation  from  the  standards  occurred,  the  operating 
procedures,  training  programs,  or  the  operational  envelope  involved  were 
examined.  All  of  the  recorded  data  and  several  calculated  items  were 
produced  in  the  form  of  a.  Hxinted  list,  known  as  a frame-by-frame  printout. 

An  underlying  assumption  of  the  Astrolog  program  was  that  an  excursion 
outside  the  established  operational  envelope  is  a warning  of  possible 
trouble,  while  operation  inside  the  established  range  is  demonstrated  proof  of 
satisfactory  performance.  The  validity  of  this  assumption  has  been  proven  by 
information  obtained  by  the  analysis  of  aircraft  accidents  and  incidents.  It 
is  also  compatible  with  our  concept  of  measuring  pilot  performance  in  an 
age-related  functional  framework. 

To  assist  the  analysis  of  data  further,  or  the  study  of  a particular 
portion  of  flight,  a third  form  of  output  was  obtained  by  Astrolog.  These  are 
profiles  of  selected  data  drawn  by  a plotting  machine.  A sample  plot  is  shown 
as  Figure  14,  This  particular  plot  is  a time  history  of  several  data  items. 
Various  types  were  available,  drawn  to  scales  appropriate  to  the  study  of 
takeoffs,  landings,  or  entire  flights  (30). 


The  recorded  data  are  also  available  for  purposes  other  than  flying 
safety  evaluation.  Other  possible  uses  include  engine  and  airplane  performance 
measurement,  automatic  tracking  for  air  traffic  delay  data,  and  analysis  of 
compliance  with  optimum  flight  plans.  In  this  context,  the  system  can  be  used 
to  record  aircraft/pilot  interaction,  and  it  yields  objective  measurements  of 
pilot  performance.  The  "Astrolog"  system  was  invented  by  Captain  W.  A. 
Braznell,  American  Airlines.  The  program  was  discontinued  in  1971  when  the 
BAC-lil  aircraft,  in  which  the  system  had  been  installed,  were  taken  out  of 
service. 


Another  example  of  an  attempt  to  make  use  of  existing  technology  for 
recording  and  assessing  pilot  performance  automatically  is  the  development  and 
application  of  the  advanced  inflight  monitoring  system  designed  by  Trans  World 
Airlines  (1WA),  Incorporated  (34).  Since  1968,  TWA  has  undertaken  to  monitor 
each  approach  and  landing  made  by  its  crew  members  during  their  routine  flight 
conditions.  In  September  1975,  TWA  recorded  the  two  millionth  monitored 
approach.  An  expanded  inflight  system  was  recently  installed  in  the  L 1011 
aircraft.  Rather  than  recording  only  seven  parameters  associated  with  the 
landing  approach,  the  new  system  records  30  flight  performance  parameters 
throughout  the  entire  flight  range  from  engine  start  to  engine  shutdown.  A 
detailed  listing  of  the  13  trend  modes  and  the  30  performance  parameters  to  be 
recorded  is  given  in  Appendix  A (see  also  Appendix  A,  Figure  A-l). 


41 


42 


Astrolog  Inflight  Monitor. 


Data  handling  and  analysis  have  been  very  well  organized.  The  data  are 
taken  on  magnetic  tape,  which  is  removed  at  layover  points  and  then  transmitted 
to  TWA's  Kansas  City  computer  via  data  terminals  and  telephone  lines.  Any 
deviation  from  the  limits  established  for  the  30  flight  crew  performance 
parameters  is  recorded  by  the  computer  along  with  the  flight  number,  date,  and 
crew.  Thus,  each  pilot's  performance  is  monitored  during  each  flight  by  an 
impartial  recorder  and  the  results  are  retained  for  later  evaluation. 

Details  concerning  the  TWA  AIDS/Inflight  Monitoring  System  are  given  In 
Appendix  A.  In  a brief  entitled  "Trend  Modes"  the.  modes  are  listed  in  which 
aircraft  operations  are  sensed  and  recorded.  There  are  three  different 
reports  generated  when  the  system  is  in  operation.  Examples  of  these  reports 
are  also  given  in  Appendix  A.  The  first  is  an  Exception  Report  obtained  as  the 
result  of  a "L  1011  Flight  Analysis."  It  contains  information  about  the 
route,  flight  crew,  takeoff  and  landing  weight  of  the  aircraft,  date,  time, 
and  mode  of  the  flight  as  well  as  type  of  exceedence  (localizer,  glide  slope, 
calibrated  airspeed,  and  descent  rate  deviations)  (See  Appendix  A,  Figure  .4-2) . 
The  parameters  listed  in  columns  7 through  11  in  this  report  show  the 
recorded  values  for  the  localizer,  magnetic  heading,  radio  altimeter,  flaps, 
and  glide  slope  deviations. 

The  second  report  is  the  "L  1011  Performance  Summary  by  Captain"  which 
contains  information  such  as  the  total  number  of  crew  performance  deviations 
during  the  entire  month,  the  total  number  of  flight  legs  monitored,  instrument 
approaches,  instrument  approach  exceedences,  and  the  number  of  exceedences  per 
flight  leg  (see  Appendix  A,  Figure  A-3).The  third  report  is  the  "L  1011  Monthly 
System  Summary"  which  provides  operational  trends  and  points  out  areas  of 
particular  concern  (see  Appendix  A,  Figure  A-4) . For  example,  exceeding  V'2  by 
more  than  15  knots  consistently  would  need  a closer  observation  and  corrective 
action.  TWA  is  convinced  that  this  program  will  increase  the  safety  of  the 
operation  and  will  provide  more  reliable  and  accurate  performance  and 
proficiency  measures  than  the  occasional  observation  in  a stereotyped 
situation  and  by  subjective  judgment. 

One  has  to  consider,  of  course,  some  of  the  shortcomings  or  weaknesses  of 
the  fully  automated  performance  measurement  method,  that  have  been  pointed  out 
by  several  investigators  (4,12,25).  First,  it  has  been  mentioned  that  auto- 
matic recordings  of  pilot  performance  does  not  show  nor  explain  what  is  going 
on  in  the  pilot's  brairi.  There  are  many  subtle  aspects  of  judgment  and 
decision  making  that  oo  not  lend  themselves  to  recording;  and  automated 
performance  measurement  usually  permits  the  assessment  of  only  those  actions 
by  the  pilot  which  directly  affect  the  performance  and  motion  of  the  aircraft. 
Hence,  a sudden  deviation  from  the  glidepath  or  an  unprogramed  increase  in 
speed  may  be  caused  by  an  unprogramed  event,  such  as  an  unexpected  obstacle  on 
the  runway,  a failure  in  the  lighting  system,  or  a visual  illusion.  And  the 
reason  for  the  "unuesired"  deviation  from  the  program  may  not  become  obvious 
from  the  records  odI aineo  during  the  pilot  action,  although  the  deviations 
were  necessitated  for  safety  reasons.  Moreover,  there  may  be  psychological  or 
psychosocial  problems  that  affect  pilot  action  and  express  themselves 
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unconsciously  and  remain  unexplained  and  may  influence,  only  temporarily  and 
with  no  lasting  degradation,  his  performance.  These  factors  can  become 
important  and  some  of  them,  in  particular  those  generated  by  the  environment 
in  flight  and  observable  to  the  inspector  pilot,  may  be  detected,  explained, 
and  analyzed  by  a subjective  assessment  technique.  By  and  large,  however, 
these  flaws  of  the  automated  objective  method  do  not  diminish  the  overall 
value  of  this  method,  which  provides  data  free  from  personal  bias. 

As  a remedy  for  the  possible  negative  features  of  the  automatically 
recording  objective  assessment  system,  a multivariate  method  has  been 
recommended  by  several  scientists  in  this  country  and  abroad  (4,25).  They 
suggest  that  subjective  ratings,  physiological  recordings,  and  automated 
measurements  be  combined  to  yield  a total  performance  score.  However,  this 
approach  also  has  some  inherent  flaws,  in  particular  since  it  is  not  always 
possible  to  attain  these  three  scores  concomitantly.  Moreover,  the 
physiological  data  obtained  under  test  conditions  are  often  ambiguous,  and 
they  may  contribute  more  uncertainties  and  variance  than  improve  reliability. 
For  certain  conditions  of  performance  measurement,  for  example,  during  solo 
flights  where  there  is  no  instructor  pilot  in  the  cockpit,  the  automatic 
recording  seems  to  be  the  only  accurate  and  reliable  means  to  collect 
performance  data,  and  in  this  case  the  recording  of  some  physiological 
parameters  can  help  to  assess  performance. 

Knoop  and  Welde  (26)  suggested  hat  pilot  acceptability  becomes  a rather 
important  point,  when  the  time  arrives  for  making  the  decision  to  implement 
an  automated  pilot  performance  measurement  system.  Apparently,  there  is 
evidence  that  pilots  accept  such  a system  if  it  has  been  proven  to  be  sensi- 
tive, valid  and  fair  (34).  As  far  as  the  training  situation  is  concerned,  it 
can  be  argued  that,  whatever  type  or  level  of  sophistication  of  advanced 
performance  measurement;  is  attained,  the  human  observer  should  always  be  part 
of  the  system.  But  this  is  not  the  point  here.  The  purpose  of  this  survey 
was  to  find  out  whether  or  not  there  exist  objective  methods  which  can  be 
used  to  obtain  performance  profiles  usable  for  the  assessment  of  pilot 
proficiency.  This  question  can  be  answered  affirmatively. 

Summary  and  Conclusions. 

The  purpose  of  this  study  was  to  describe  how  pilot  performance  can  be 
monitored  and  assessed,  and  what  means,  techniques,  methods,  and  instruments 
are  available  to  measure  pilot  performance  accurately  and  reliably.  Such 
measurements  will  have  to  be  made  if  a functional  age  index  or  an  objective 
proficiency  standard  for  pilots  is  to  be  developed  that  can  be  used  as  a 
criterion  for  extending  or  terminating  an  aviator's  career. 

It  has  been  shown  in  this  context  that  the  attempt  to  develop  criteria 
and  means  for  the  assessment  of  pilot  performance  dae^c  back  to  World  War  I. 
There  were  two  major  approaches  taken  in  order  to  reach  this  goal;  namely, 

(i)  the  qualitative  evaluation  of  performance  based  mainly  on  instructor 
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ratings  and  flight  inspector  Judgments,  and  (ii)  quantitative  grading  of 
performance  based  on  numerical  rating  scales  and  recordings  of  pilot  actions 
which  reflect  the  quality  of  the  performance.  Several  examples  were  given  to 
illustrate  these  efforts. 

Within  the  qualitative  assessment  system,  which  is  highly  subjective  in 
nature,  there  are  several  steps  of  sophistication,  ranging  from  a simple 
pass/fail  rating  to  detailed,  muiti-facet?d  descriptions  of  the  examinee's 
behavior,  personality,  and  performance.  It  has  been  voiced  by  many 
researchers  familiar  with  psychological  assessment  techniques  that  any  attempt 
at  manually  recording  inflight  activities  is  highly  questionable,  since  the 
rater  is  often  unable  to  effectively  time-share  the  task  of  observing  and 
recording  multiple  parameters  at  an  appropriate  sampling  rate.  His  judgment, 
primarily  based  on  ar  overall  impression  of  the  examinee's  effort,  may  be 
involuntarily  biased,  unreliable,  and  occasionally  unfair.  Actually, 
however,  this  method  is  still  being  used  and  is  generally  accepted  and 
operationally  rather  effective. 

The  more  advanced  method  of  measuring  pilot  performance  is  based  on  the 
concept  that  data  should  be  recorded  objectively  and  independently  of  the 
ability,  judgment,  and  standards  of  the  examiner/inspector.  The  highest 
degree  of  accuracy  and  reliability  can  be  attained  when  permanent  records  of 
actions  and  behavior  of  the  pilot  are  furnished  by  an  automated  data 
acquisition  system.  Review  of  the  pertinent  literature  suggests  that  the 
following  steps  are  indicated  in  the  development  and  use  of  an  objective 
performance  measurement  system: 

1.  Performance  analysis  in  order  to  establish  quantifiable 
descriptors  or  criteria  of  performance  (including  the 
definition  of  errors,  scales,  and  scoring  techniques). 

2.  Raw  data  collection. 

3.  Selection  of  a unit  of  measurement  in  regard  to  human  subsystem 
or  operator  performance. 

4.  Selection  of  the  important,  adequate,  and  useful  measurable 
parameters. 

5.  Measurement  System  test  and  evaluation. 

6.  Calibration  and  standardization  of  the  measurement  system  and 
its  validation  against  the  intended  purpose  and  other  available 
modes. 

7.  Calibration  and  standardization  of  the  data  and  preparation 
of  the  information  In  a practical,  manageable,  and  usable 
form. 
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It  was  shown  in  the  course  of  this  discussion  that  with  all  the  computers 
and  ADP  available  today,  pilot  performance  cai,  be  measured  objectively, 
accurately,  and  reliably.  Such  measurements  discriminate  effectively  between 
different  levels  of  operational  requirements,  demands,  skill,  arid  proficiency. 
If  properly  evaluated,  such  data  should  be  useful  not  only  for  measuring  pilot 
performance  at  a particular  point  in  time,  but  also  for  predicting  later  or 
expected  proficiency  through  the  analysis  of  current  performance  and  its 
comparison  with  past  performance. 

The  military  services,  private  industry,  and  the  airlines  have  made  great 
.strides  in  the  design  and  application  of  objective,  automatically  recording, 
inflight  monitoring  systems.  While  mostly  developed  for  research  purposes, 
they  are  now  being  viewed  for  application  on  a routine  and  regulatory  basis. 
Owing  to  their  capability  of  monitoring  simultaneously  the  performance  of  the 
aircraft  and  the  human  operator,  they  are  the  ultimate  in  assessment  systems 
design  and  application.  They  offer  great  possibilities  for  the  establishment 
of  a functional  age  index  for  pilots.  Most  probably,  this  development  will 
first  affect  the  air  carriers;  but  the  other  groups,  namely,  the  military  and 
the  general  aviation  pilots,  will  also  utilize  the  advantages  offered  by 
progress  in  this  area.  The  vertification  of  the  concept  and  its  validation 
is  still  a matter  of  future  research. 
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APPENDIX  A 


TWA  INFLIGHT  MONITORING  SYSTEM 
TREND  MODES 


Trend 


Mode  No. 

Title 

Description  of  Cue  Initiation 

T 1 

ESU 

Engine  Start  - No.  1 , 2 or  3 Fuel/Ignition 

Switch  On. 

T2 

TKOR 

Takeoff  Roll  - No.  1 Engine  Thrust  Lever 
advanced  to  70%  power. 

T3 

V2 

Radio  Altitude  > 35  Feet. 

T4 

CLB1 

Climb  1 - Radio  Altitude  2 1, 600  Feet. 

T5 

CLB2 

Climb  2 - Altitude  Coarse  .2  9,  £55  Feet, 

T6 

CEE  3 

Climb  3 - Altitude  Coarsei  12,000  Feet. 

T7 

CRZ 

Cruise  - Pitch  Computer -Altitude  Hold  Mode 
is  engaged  for  15  minutes. 

T8 

DST1 

Descent  1 - Pitch  Computer -Altitude  Hold 

Disengage  and  Altitude  Coarse  Decreases.?  1,  OOOr  i, 

T9 

DST2 

Descent  2 - Altitud.  Coarse  £ 9,450  Feet. 

T10 

APP1 

Approach  - Radio  Altitude  - 1,500  Feet. 

Til 

APP2 

Approach  2 - Radio  Altitude  E 500  Feet, 

T12 

ROLT 

Rollout  - Air/Ground  Sensor  - Aircraft  on  ground. 

Ti  3 

ESD 

Eng;ne  Shutdown  - No,  1,  2 and  3 Fue}./ Ignition 
Switches  Off. 

Logic  is  provided  for  alternate  flow  of  trend  mode  regression  as  indicated 
on  the  following  chart.  Trend  mode  cue  initiation  is  the  same  as  above. 


r LIGHT  CgKW  PERIORMANI*  PAR  AM £ TC  R5 


2,  3. 4, 5. 6. 
7,  tf.9,  tO,  11 


2.  1.4.  5.6. 
7,  8,  9, 10.  U 


J.  4.  5.  o,  7. 
8.  V,  I 0,  1 l 


4, 5.0, 7, a. 
9,  10,  11 


DsscrWijn  (SuppoK  t>aramim_d*Hi  W fr«  Bf^Kd)  ■ £& 

At  th#  start  of  the  txkaoff  roll.  flaps  shall  ba  ««l  at  1' 

19°  and  pilch  trim  ah*  11  be  set  wtlhla  -2,  5°  Co  *8°. 

(Hitch  trim,  C.G, , gross  weight,  flaps.) 

Whan  the  Radio  Altimeter  Altitude  equal*  15  *'••! 

(V^),  computed  air  speed  ehall  b*  less  than 
knot*  and  greater  thin  Vj  -5  knots  (CAS) 

At  a Radio  Altimeter  Altitude*  ©/  200‘,  pitch  attitude 
• halt  be  lea*  man  19w  and  greater  than  Jl°.  [CAS. 
praaaure  altitude,  radio  altitude,  pilch  attitude. ) 

Roll  attitude  ehall  be  l««r  than  35  and  treat  r than 
-J5°.  (CAh,  pressure  altitude,  radio  altitude, 
roll  attitude. ) 

Vertical  accelei alien  ehall  be  loi  than  1. 5G‘a  and 
greater- than  0.  5G's.  (CAS.  preiiure  altitude,  radio 
altitude,  vertical  accchrUun, ) 2: 

The  angle  of  attach  ehall  be  tea*  (hnn  18.  3 . (CA^ 
preature  altitude,  A.  O.  A.  , pilch  attitude. ) 

Trend  Made  3 (V,)  shall  bv  Minted  on*  minute  alter 

(h«  etart  ol  the  ukc«(l  roll.  (Word*:  Abort  Takeoff.)  {] 

Computed  air  speed  ehall  o«  ■'iihn  the  following  limits: 

Ftapa  10°  - CAS  aliall  be  It-* a than  240  Knot*  and 
greater  than  -5  knot*. 
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Flap*  4°  - CAS  Shall  bo  lens  than  250  knot*  and 
greater  ih.m  V»  *5  knot*, 

Flap*  0°  . CAS  shall  b»  leas  than  2(.0  knot*  *-id 

greater  than  * 50  knot*.  2 : 

(CAS.  pr<  -sure  altitude,  radio  altuudc,  (lap*.) 

Whenever  th»-  landing  gear  i*  being  retraced, 

compute <1  air  ipfi-d  s)-aU  be  h: » s than  240  knot*.  ^ 

(CAS,  prtssu*  i-  altitude.  > 

When  tin-  landing  gur  btart*  to  retract,  the  rale  of 
t limb  shall  bi- positive,  (CAS.  pressure  altitude.  2' 

radio  altitude,  pitrh  attitude.  Word*:  Alt  Fine  -Oecr.  1 

Filch  altitude  shall  b*-  lea*  than  19°  and  greater 

i|, an  -i~.  (CAS,  pi c»*ur.' altitude,  pitch  altitude,  2< 

Capa.  1 

During  the  takcofl  roll  unlil  ll/lofl  ard  dur'nt  approach 
below  .i  radio  altimeter  altitude  of  50  feist,  pitch 
attiimh  alia II  he  less  than  12.5°.  (CA5,  pitch  attitude, 

radio  altitude,  flap*.  Vertical  acceleration. ) 2; 

From  a pressure  altitude  ot  14,000  feci  to  28,900 
fit”.,  torn  puled  air  speed  shall  less  than  JJ5  knot* 
and  greater  than  305  knot*.  (CAS,  pressure  altitude.) 

At  a pressure-  altitude  greater  than  28,400  ivvi- 
M-'ih  numb i r shall  be  iis>  than  0.85  and  gr >attr 
than  0.  79.  (Mach  nunih'-r,  pressure  altitude.  ) 
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In  th«  Cruise  Mode  at  |>ri'>iurt  altitude*  greater  than 
3d.  000  .‘cel,  Mach  number  ahull  be  lee*  *lian  0.87  ami 
greater  than  O.  el . (Mach  number,  pressure  alG'udv, 
•tltitud'  hold«V»  »/tio. ) ,r 

rin  re  shall  uv  no  ’Altitude  Scion*  deviation  indication* 
for  more-  than  tv.  o second*.  (CAS,  pressure  altitude. 
rP'tio  altitude,  * 

A TOR  deviation  greate  r than  one  dot  lor  a period  of 
ten  minutes  nhill  include  a heading  change  ol  greater 
than  15°.  (CAS,  pressure  altitude,  VOR  deviation, 
magnetic  heading,  Mach  number.) 

flu  ring  thi  descent  mode.  Moth  number  shall  bn  less 
than  0.9'ii  and  computed  air  sp**d  shall  be  let*  than 
J7’’knot».  (CAS,  pressure-  altitude,  Mach  number. ) 


_ ^ Pascrlpiljp  (Support  yarams'sr  data  to  be  prjntsd} 

*■«  10«  >1  Computed  air  speed  sliall  be  within  the  following  limits: 

Flap*  w°  - C AS  shall  l*r  l«i«  than  Z60  knots  and 
greater  tnan  Yr«l  ♦ 50  knots, 

Flaps  4°  - Ca5  shall  U<  hu  than  250  kuots  and 
grratsr  than  Vraf  + 20  knots. 

Flap*  10*  • CAS  ehall  b*  lesa  than  2 40  knote  and 
greater  than  Vief  ♦ 19  knots. 

Flaps  22°  - CAS  shal:  b«  tc*e  than  203  knote  and 
greater  than  Vr*l  ♦ i knots. 

Flaps  31°  - CAS  *hall  be  lees  than  170  knots  and 
greater  than  Yrcf  - 4 knots,  (CAS, 
pr***ure  altitude,  radio  altitude,  flaps.  | 

11  from  ■ radio  altimeter  altlfuJe  of  500  feet  to  100  feet, 

1<  call ser  \nd  glide  slope  deviation*  *ha”.  be  less  than 
One  dot.  (CAS,  Isislue:  deviation,  radio  altitude, 
flaps,  glide  elope  deviation,  gross  weight,  magnetic 
hm  ling,  | 

11  From  a radio  altimeter  altitude  of  500  feet  to  touch- 

down, the  rate  of  descent  computed  aver  b ala  second 
period  shall  be  les,  than  900  feet  per  rptnut:.  (CAS. 
pressure  altitude,  radio  altitude,  desv'nt  rate, 
magnetic  heading.) 

11  Ironi  a radio  Mill  leier  altitude  of  $00  feet  to  touch- 

down, the  rat of  desc.  nt  computed  over  a six  second 
period  shall  b«-  positive.  (CAS,  pressure  altitude, 
radio  altitude,  flaps,  magnetic  heading. ) 

11  Front  a radio  altimeter  altitude  of  SO1’'  fist  to  20  feet, 
computed  air  .peed  shaii  b«  ire#  than  Vrrf  ♦ lo  knots 
and  greater  than  Vref  ■ 5 knots.  (CAS,  pressure 
altitude,  radic  vltitudc,  flaps,  great  weight. ) 

3.4,  5.  t,  7,  While  eh*  Landing  gear  la  down,  computed  air  speed 

8,  9,  1 0,  1 1 shall  be  iris  than  250  knot#  «nd  Mach  number  shall  b'. 

lc*a  than  0.  7 J.  (CAS,  pressure  altitude,  Mach  number. 

«l  The  time  from  s radio  shim  w altitude-  ol  50  leet 

to  touchdown  shall  be  less  than  15  seconds  ('.“AS, 
touchdown  -y ci/no.  radio  altitude. ) 

J,  1.5.6,  *,  Ai  cxcccdancc  r <■  r ordir.g  will  rot-.-mcnce  if  the  ground 
8.9.10,11  proximity  pull -up  Sight  i*  illuminated  ,<nd  the  ground 

proximity  fault  nghr  is  extingu.xhcd.  (CAS,  pressure 
alti’udv,  radio  altitude,  flaps,  Hiord*;  CPWS  pull  uj>. 
g ca r dow n - yea'no.  ) 

12  Hsrd  landing  indications  shall  not  exceed  1)7,(»00 

pounds  for  .'he-  left  and  right  main  gear*  and  105.000 
pound*  foi  th-  n»kt|.4j,  (CA5,  left  main,  right  man., 
nose,  pitch  atitudc,  roil  etitudt,  vertical  acceleration.  | 

12  The  t.me  to  spoiler  action  after  touchdown  shall  h« 

loss  than  5 vecontJa  fvAS,  touchdown  - yes/no, 
spoiler  - yes/no.  ) 

12  Th'  iimi-  to  reverse  thrust  action  cn  any  engine  after 
touchdown  shall  be  less  than  7 seconds.  (CAS,  touch- 
down - yes  f no,  thrust  revi-ri.  - yus/no.) 

12  F rent  louchdown  to  touchdown  *■  12  seconds,  brake 
m.-tvred  hydraulic  presaurc  .*hbll  be  less  than  1 OQO 
1351.  (CAS.  touchdown  - yea/no,  brakes.) 


Additionally,  the  AIUS  Oata  Ln* ry  Panel  has  b<-rn  modified  to  include  a system 
selection  labc-kd  ’Instrumented  Appioach1.  Wlienthi*  porition  It  selected  by 
lit v 1 light  Crew,  pnu-  to  1 500  fc-it  radio  tilimclir  altitude,  and  the  manual 
rc-iorii  Du. ion  is  d.  presse  d,  tin  on -board  computer  will  Mag  the  data  recorded 
ofu  r this  uvi-m  at  bi-ir.g  on  instrument  approacu, 
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EXCEPTION  REPORT 

LIOH  PERFORMANCE  SUMMARY  BY  CAPTAIN  (MONTHLY) 
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Figure  A- 5 (Continued) 


